# README reward\_evolution_data.json

This file consists of data describing the progress of internal reward evolution of the agents (used as a basis for Figure 4).

Keys:

* ```DEFAULT_REWARD_CONFIG``` - mapping between internal reward ID and description of event that generates reward.
* ```ordered_by_type_per_step``` - dictionary where keys are training steps, values are internal reward values at that step.
* ```reward_type_to_step_value``` - same data as ```ordered_by_type_per_step``` but with keys being the reward ID and the values being dictionaries with steps and reward values.

## Example code

    import json
    import numpy as np
    import matplotlib.pyplot as plt

    with open("reward_evolution_data.json") as fh:
      rewdata = json.load(fh)

    # Mapping of reward ID to descriptions
    reward_config = rewdata['DEFAULT_REWARD_CONFIG']

    # Steps during training
    step_rewards = sorted([(float(s), v) for s, v in rewdata['ordered_by_type_per_step'].iteritems()])

    plt.figure()
    for rewid in reward_config.keys():
      x = [s[0] for s in step_rewards]
      y = [s[1][rewid] for s in step_rewards]
      plt.plot(x, y, label=reward_config[rewid])
    plt.legend(bbox_to_anchor=(2, 1))
    plt.xlabel("Training Steps")
    plt.ylabel("Internal Reward Value")
    plt.show()


# README ctf_learning_curves.json

This file consists of data describing the learning progress of agents (used as the basis for Figure 2). This is the Elo for the agents and humans.
Each key in a dictionary is one entry from the Figure, and consists of a dictionary with "games played" (used for x-axis) and "Elo" (used for y-axis)

## Example code

    import json
    import matplotlib.pyplot as plt
    
    with open("ctf_learning_curves.json") as fh:
      curves = json.load(fh)

    plt.figure()
    for key in curves:
      plt.plot(curves[key]["games played"], curves[key]["Elo"], label=key)
    plt.legend()
    plt.show()


# README hyperparmeters.json

This file consists of data describing the evolution of some hyperparameters.

Keys:

* ```learning_rates_by_step``` - list of tuples, each of which contain the training step as first element, and list of learning rate values for the population of agents as the second element.
* ```kl_weights_by_step``` -  list of tuples, each of which contain the training step as first element, and list of KL weight values for the population of agents as the second element.
* ```time_periods_by_step``` - list of tuples, each of which contain the training step as first element, and list of internal time period values for the population of agents as the second element.

## Example code

    import json
    import numpy as np
    import matplotlib.pyplot as plt

    with open("hyperparameters.json") as fh:
      curvedata = json.load(fh)

    # Plot hyperparamters
    hyper_parameters = ['learning_rates', 'kl_weights', 'time_periods']
    for hyper_parameter in hyper_parameters:
      hyper_data = curvedata['%s_by_step' % hyper_parameter]
      # Convert steps to float
      hyper_data = sorted([(float(s[0]), s[1]) for s in hyper_data])
      x = [s[0] for s in hyper_data]
      # Calculate mean and standard deviation for each
      # time step across agent population
      ymean = np.array([np.mean(s[1]) for s in hyper_data])
      ystd = np.array([np.std(s[1]) for s in hyper_data])
      # Plot
      plt.figure()
      plt.plot(x, ymean, 'b')
      plt.fill_between(x, ymean-ystd, ymean+ystd, color='b', alpha=0.3)
      plt.title(hyper_parameter)
      plt.xlabel("Training Steps")
      plt.ylabel("Value")
      plt.show()


# README tsne\_data.json

This file consists of data for the TSNE plot of agent LSTM activations and their associated CTF state labels (used as the basis for Figure 3).

Keys:

* ```x``` - list of horizontal positions for each point.
* ```y``` -  list of vertical positions for each point.
* ```state``` - binary vector for each point denoting which states the agent is in (1 is active, 0 is inactive).
* ```state_labels``` - text label for each state.

## Example code

    import json
    import numpy as np
    import matplotlib.pyplot as plt

    with open("tsne_data.json") as fh:
      tsne_data = json.load(fh)

    states = np.array(tsne_data['state'])
    for j in range(len(tsne_data['state_labels'])):
      plt.figure()
      plt.scatter(tsne_data['x'], tsne_data['y'], c=states[:,j], alpha=0.6, cmap='viridis', vmin=0, vmax=1, edgecolors='none')
      plt.title(tsne_data['state_labels'][j])
      plt.show()


# README behaviour\_data.json

This file consists of data for the behaviour plots (used in Figures 3 and 4).

Keys:

* ```training_episodes``` - number of episodes the agent which was measured for each data point was trained.
* ```teammate_following``` -  probability of teammate following behaviour occuring throughout training.
* ```opponent_base_camping``` -  probability of opponent base camping behaviour occuring throughout training.
* ```home_base_defence``` -  probability of home base defence behaviour occuring throughout training.
* ```average_per_episode_count``` - average per-episode count of behaviours occuring. Each behaviour has data and label.

## Example code

    import json
    import matplotlib.pyplot as plt

    with open("tsne_data.json") as fh:
      behaviours = json.load(fh)

    plt.figure()
    plt.plot(behaviours['training_episodes'], behaviours['teammate_following'], 'b')
    plt.plot(behaviours['training_episodes'], behaviours['opponent_base_camping'], 'r')
    plt.plot(behaviours['training_episodes'], behaviours['home_base_defence'], 'g')
    plt.show()

    for k, v in behaviours['average_per_episode_count'].iteritems():
      plt.figure()
      plt.barh(range(len(v['data'])), v['data'], tick_label=v['labels'])
      plt.title(k)
      plt.show()


# README knowledge\_data.json

This file consists of data for the knowledge plot (used in Figure4).

Keys:

* ```training_episodes``` - number of episodes the agent which was measured for each data point was trained.
* ```data``` - knowledge percentage at each point in training.

## Example code

    import json
    import matplotlib.pyplot as plt

    with open("tsne_data.json") as fh:
      knowledge = json.load(fh)

    plt.semilogx(knowledge['training_episodes'], knowledge['data'])

